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Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
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with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
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Reuben  Pitts — Mr.  Pitts  is  the  president  of  Lyceum  Consulting.  He  joined  the  Naval  Weapons  Lab  in 
Dahlgren,  VA,  in  June  1968  after  graduating  from  Mississippi  State  University  with  a  BSME.  His  early 
career  was  spent  in  ordnance  design  and  weapons  systems.  He  subsequently  served  on  the  planning 
team  to  reintroduce  the  Navy  to  Wallops  Island,  VA,  currently  a  multiple  ship  combat,  over-the-water 
weapons  testing  lab  for  Surface  Ship  Combat  Systems,  Fighter  Aircraft,  and  live  missile  firings.  His 
outstanding  service  as  the  deployed  Science  Advisor  to  Commander,  U.S.  Sixth  Fleet  was 
recognized  with  the  Navy’s  Superior  Civilian  Service  (NSCS)  Award  and  the  Navy  Science 
Assistance  Program  Science  Advisor  of  the  Year  Award. 

Mr.  Pitts  was  selected  to  lead  the  technical  analysis  team  in  support  of  the  formal  JAG 
investigation  of  the  downing  of  Iran  Air  Flight  655  by  USS  Vincennes,  and  participated  in  subsequent 
briefings  to  CENTCOM,  the  Chairman  of  the  Joint  Chiefs,  and  the  Secretary  of  Defense.  As  head, 
Surface  Ship  Program  Office  and  Aegis  program  manager,  Mr.  Pitts  was  awarded  a  second  NSCS, 
the  James  Colvard  Award,  and  the  John  Adolphus  Dahlgren  Award  (Dahlgren’s  highest  honor)  for  his 
achievements  in  the  fields  of  science,  engineering,  and  management.  Anticipating  the  future  course 
of  combatant  surface  ships,  Mr.  Pitts  co-founded  the  NSWCDD  Advanced  Computing  Technology 
effort,  which  eventually  became  the  Aegis/DARPA-sponsored  High  Performance  Distributed 
Computing  Program;  the  world’s  most  advanced  distributed  real-time  computing  technology  effort. 
That  effort  was  the  foundation  for  the  Navy’s  current  Open  Architecture  Initiative. 

In  2003  Mr.  Pitts  accepted  responsibility  as  technical  director  for  PEO  Integrated  Warfare 
Systems  (IWS),  the  overall  technical  authority  for  the  PEO.  In  September  of  that  year,  he  was 
reassigned  as  the  major  program  manager  for  Integrated  Combat  Systems  in  the  PEO.  In  this 
position,  he  was  the  program  manager  for  the  Combat  Systems  and  Training  Systems  for  all  U.S. 
Navy  Surface  Combatants,  including  Aircraft  Carriers,  Cruisers,  Destroyers,  Frigates,  Amphibious 
Ships,  and  auxiliaries.  In  July,  2006,  Mr.  Pitts  returned  to  NSWCDD  to  form  and  head  the  Warfare 
Systems  Department.  While  in  this  position,  he  maintained  his  personal  technical  involvement  as  the 
certification  official  for  Surface  Navy  Combat  Systems.  He  also  served  as  chair  of  the  Combat  System 
Configuration  Control  Board  and  chair  of  the  Mission  Readiness  Review  for  Operation  Burnt  Frost, 
the  killing  of  inoperative  satellite  USA  193. 

Mr.  Pitts  has  been  a  guest  speaker/lecturer/symposium  panelist  at  many  NAVSEA-level  and  DoD 
symposiums,  conferences  and  at  the  Naval  Postgraduate  School,  the  Defense  Systems  Management 
College,  and  the  National  Defense  University.  For  19  years  Mr.  Pitts  was  the  sole  certification 
authority  of  all  Aegis  Combat  System  computer  programs  for  fleet  use.  He  retired  from  the  U.S.  Civil 
Service  in  September  2008,  with  over  40  years  of  service  to  the  Navy. 
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officer  from  1990-1993  and  the  34th  support  group  director  of  security,  plans,  and  operations  from 
1986-1987.  Prior  to  that,  LTC  (Ret.)  Naegle  held  positions  in  test  and  evaluations  and  logistics  fields. 
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from  1989-1991  and  the  Detroit  Arsenal  Tank  Plant  from  1982-1984.  COL  Boudreau  is  a  graduate  of 
the  Industrial  College  of  the  Armed  Forces;  Defense  Systems  Management  College;  Army  Command 
and  General  Staff  College;  Long  Armour-Infantry  Course,  Royal  Armoured  Corps  Centre,  United 
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Abstract 

The  intent  of  this  research  is  to  gather  together  the  various  approaches  for  controlling  and 
reducing  Total  Ownership  Cost  (TOC)  and  to  describe  tools  and  methods  to  assist  PMs  and 
others  in  addressing  TOC  more  effectively.  This  study  examines  TOC  from  the  perspective  of 
congressional  direction,  the  perspective  of  the  OSD  and  Service  leadership’s  governance, 
the  perspective  of  PM  execution,  and  the  perspective  of  available  infrastructure  support. 

Purpose 

The  intent  of  this  research  is  to  gather  together  the  various  approaches  for 
controlling  and  reducing  Total  Ownership  Cost  (TOC)  and  to  describe  tools  and  methods  to 
assist  PMs  and  others  in  addressing  TOC  more  effectively. 

Scope  of  This  Study 

This  study  examines  TOC  from  the  perspective  of  congressional  direction,  the 
perspective  of  the  OSD  and  Service  leadership’s  governance,  the  perspective  of  PM 
execution,  and  the  perspective  of  available  infrastructure  support. 

Introduction 

This  report  extends  our  research  that  was  first  published  in  2003.  At  that  time,  just  as 
currently,  there  was  significant  attention  being  paid  to  TOC.  There  were  a  number  of 
initiatives  collected  and  shared  on  a  TOC  website  constructed  by  the  Institute  for  Defense 
Analyses  (IDA;  www.ida.org).  Additionally,  the  DAU  Acquisition  Community  Connection 
website  (https://acc.dau. mil/CommunityBrowser.aspx?id=22509&lang=en-US)  also  contains 
useful  approaches  to  TOC  and  R-TOC.  Looking  over  the  TOC  landscape  in  2003,  one 
would  not  conclude  that  there  was  a  shortage  of  ideas  related  to  reducing  TOC.  The  same 
appears  true  today — there  are  many  useful  approaches  for  reducing  TOC,  or  weapon 
system  life  cycle  costs,  reflecting  the  increasing  anxiety  over  skyrocketing  costs  of 
ownership.  Many  aspects  of  Defense  acquisition  have  continued  to  evolve,  making  it  difficult 
to  know  what  has  helped  to  control  costs  and  what  may  have  had  the  opposite  effect  or  had 
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no  significant  effect.  The  following  paragraphs  provide  a  few  examples  to  help  make  the 
point. 

There  are  increased  acquisition  reviews  (USD[AT&L],  2008).  PMs  and  those  working 
in  program  offices  know  that  reviews  are  expensive  and  divert  attention  from  other 
management  activities.  Have  increased  reviews  contributed  to  increased  cost  or  have  they 
reduced  it?  Has  developmental  cost  increased  while  the  larger  sustainment  costs  have 
decreased?  Does  anyone  really  know? 

Acquisition  reforms,  launched  in  the  mid-1990s,  resulted  in  many  changes  to  the  way 
we  do  acquisition  business.  For  example,  acquisition  programs  have  reduced  their 
preparation  for  sustainment.  MIL-STD-1388-2A  and  -2B,  which  became  obsolete  under  the 
Acquisition  Reform  initiatives  of  the  1990s,  were  very  detailed  and  for  many  years  had 
guided  acquisition  logistics  planning;  they  were  mandatory  until  circa  1995.1  These 
standards  governed  supportability  analyses  and  served  to  inform  sustainment  planning,  but 
they  were  onerous  requirements  and  sometimes  resulted  in  analyses  that  languished  on  the 
shelf  and  were  never  put  to  use.  Did  the  discontinued  use  of  these  standards  result  in  the 
de-emphasis  and  de-funding  of  rigorous  sustainment  planning,  in  turn  causing  an  increase 
in  the  cost  of  sustainment  and  a  corollary  reduction  in  warfighting  system  readiness? 

Another  acquisition  reform  initiative  during  the  mid-1990s  created  a  bias  against 
purchasing  technical  data  packages  (TDPs).2  Did  that  result  in  the  avoidance  of 
unnecessary  and  unneeded  TDPs,  or  might  this  initiative  have  prevented  the  purchase  of 
technical  data,  leaving  a  program  with  few  good  options  related  to  re-buys  and  purchase  of 
repair  parts?  Did  it  narrow  the  range  of  choices  related  to  component-  and  system-level 
maintenance? 

Has  performance-based  logistics  (PBL) — mandated  in  the  DoD  by  the  QDR  in 
September  2001  and  implemented  in  2002  (USD[AT&L],  2002) — reduced  the  cost  of 
sustainment  or  has  it  increased  those  costs?  Coupled  with  early  tech  data  choices,  have 
logisticians  been  forced  into  choices  that  make  sustainment  more  expensive  throughout  the 
weapon  system’s  life  cycle  (Kratz  &  Buckingham,  2010)? 

First  Gut  Question 

Have  Acquisition  Reform  and  Acquisition  Excellence  initiatives  removed  acquisition 
controls  and  opened  up  an  array  of  poor  choices  for  PMs  that  have  increased  system  life 
cycle  costs  (LCC)?  Might  well-meaning  Acquisition  Reform  and  Acquisition  Excellence 
initiatives  have  offered  shortcuts  that  have  ended  badly  (Kratz  &  Buckingham,  2010)? 

Second  Gut  Question 

Has  one  of  the  principal  problems  been  lack  of  discipline?  In  our  2003  paper 
(Boudreau  &  Naegle,  2003),  we  addressed  leadership  resolve  and  the  need  to  speak  with 
one  voice  about  affordability.  In  2003,  the  new  JCID’s  directives  did  not  emphasize 
affordability.  Today  those  directives  do  (for  example,  CJCS,  2009a,  Enclosure  A,  paragraph 
2-b  and  Enclosure  B,  paragraph  3-d;  CJCS,  2009b,  Enclosure  G,  paragraph  1-d  and 
Appendix  A  to  Enclosure  G,  paragraph  16;  Weapon  System  Acquisition  Reform  Act 


1  In  the  mid-1990s,  there  were  numerous  Acquisition  Reform  initiatives  intended  to  streamline  acquisition 
processes  and  reduce  cost.  One  of  these  initiatives  was  “specs  and  standards”  reform.  Many  government  specs 
were  rescinded  to  reduce  the  government  burden  and  cost  of  maintaining  specs;  in  many  cases,  the  government 
switched  to  commercial  specifications  that  were  maintained  by  various  technical  societies  or  associations.  Other 
mandatory  specs  were  rescinded  because  they  were  thought  unnecessary  or  provided  insufficient  benefit  for  the 
cost  expended.  MIL-STD  1388-2A  and  -2B  were  thought  by  some  to  fall  into  the  latter  category. 

2  Another  Acquisition  Reform  initiative  was  avoiding  the  purchase  of  technical  data  packages  in  support  of  new 
systems. 
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[WSARA],  2009,  §  201).  Yet  one  must  ask,  do  user  study  groups  understand  their  emerging 
system’s  slice  of  mission  area  funding  over  its  life  cycle?  Do  users  take  ownership  control  of 
these  costs  by  establishing  key  performance  parameters  (KPPs)  or  key  system  attributes 
(KSAs)  for  O&S  cost  or  system  life  cycle  cost?  Do  SoS  and  net-centric  system  PMs 
understand  and  account  for  TOC  drivers  associated  with  system  changes  (especially 
software)  that  impact  system  platforms  and  platform  changes  that  impact  overarching 
systems?  Do  materiel  developers  insist  on  clear,  unambiguous  sustainment  cost  goals  and 
establish  solid,  well-reasoned  CAIV  targets?  Do  contractors  structure  their  developments  to 
deliver  warfighting  systems  that  meet  customer  cost  constraints?  A  dominant  problem  might 
be  discipline — cost  discipline — starting  with  the  OSD  and  Service  leadership  and  including 
users,  materiel  developers,  and  contractors. 

Third  Gut  Question 

Is  ownership  cost  data  being  collected  and  placed  in  databases  that  facilitate 
analysis  and  comparison  to  ownership  cost  targets  such  that,  program  by  program, 
interested  parties  can  see  whether  DoD  programs  are  performing  within  their  affordability 
constraints?  Acquisition  leaders  must  be  able  to  measure  cost  performance.  If  they  really 
want  to  get  TOC  under  control,  O&S  cost  must  be  sufficiently  accurate  and  detailed  that  it 
can  be  used  to  suggest  where  system,  subsystem,  or  component  improvements  are 
needed. 

Congressional  Intervention 

Interestingly,  the  questions  posed  above  appear  to  have  been  congressional 
questions,  too.  Congress  already  seems  to  have  responded  to  an  array  of  similar  concerns, 
in  its  own  unique  way.  This  is  what  the  WSARA  of  2009  is  all  about.  This  is  what  Congress 
is  addressing  in  its  changes  to  Nunn-McCurdy.  This  is  what  motivated  Congress  to  require 
certificates  at  Milestones  A  and  B  (10  U.S.C.  §  2366a,  b).  This  appears  to  be  the 
congressional  motive  in  Public  Law  111-84  (National  Defense  Authorization  Act,  2009), 
which  institutes  product  support  managers.  Having  witnessed  a  lack  of  cost  and  process 
discipline  spanning  many  years,  particularly  in  the  area  of  sustainment  costs,  Congress  has 
acted  to  enforce  discipline,  instituting  procedures  with  force  of  law  to  get  weapon  system 
costs  under  control. 

The  Weapon  System  Acquisition  Reform  Act  of  2009 

The  Weapon  System  Acquisition  Reform  Act  of  2009  is  a  congressional  initiative  to 
increase  rigor  in  development  of  DoD  Major  Defense  Acquisition  Programs  (MDAPs).  The 
principal  intent  seems  directed  at  controlling  the  ownership  cost  of  the  DoD’s  warfighting 
systems.  The  WSARA  advances  on  a  number  of  different  fronts,  as  follows. 

The  WSARA  named  a  series  of  appointive  positions  in  the  Office  of  the  Secretary  of 
Defense  (SECDEF)  that  would  have  key  authorities  and  responsibilities  in  controlling  the 
acquisition  process.  One  such  position  is  the  Director  of  Cost  Assessment  and  Program 
Evaluation  (Director  CAPE),  who  has  major  responsibilities  in  the  areas  of  cost  estimating, 
cost  analysis,  and  advice  in  planning  PPBE,  advising  the  JROC,  and  formulating  study 
guidance  used  to  conduct  analysis  of  alternatives  of  new  major  defense  acquisition 
programs.  These  responsibilities  place  the  Director  CAPE  in  a  position  to  provide  advice 
and  direction  related  to  the  accuracy  of  acquisition  cost  estimates  and  the  affordability  of 
acquisition  programs.  The  Director  CAPE  is  specifically  charged  by  Congress  with  ensuring 
the  accuracy  of  cost  estimation  and  cost  analysis  by  prescribing  policies  and  procedures 
specifically  related  to  acquisition  programs.  The  Director  CAPE  provides  guidance  to  and 
consults  with  OSD  leadership  and  the  secretaries  of  the  military  departments  regarding 
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specific  cost  estimates  and  cost  analyses  to  be  conducted  for  a  major  MDAP  or  major 
automated  information  system  (MAIS)  program. 

JROC 

The  WSARA  specifically  charges  the  SECDEF  to  ensure  that  the  JROC  is  engaged 
in  consideration  of  trade-offs  among  cost,  schedule,  and  performance  objectives  (§  201).  It 
was  noted  in  our  2003  R-TOC  report  that  the  JROC  was  not  focused  on  TOC  and  that  the 
leadership  was  not  “speaking  with  one  voice”  concerning  the  importance  of  TOC  (Boudreau 
&  Naegle,  2003,  p.  49).  This  now  appears  to  have  been  addressed  as  a  matter  of  law. 

Milestone  Decision  Authority  (MDA) 

The  WSARA  mandates  that  MDA  ensure  appropriate  trade-offs  among  cost, 
schedule,  and  performance  objectives  to  increase  confidence  that  the  program  is  affordable 
(WSARA,  2009,  §  201). 

Competition  Throughout  the  Life  Cycle 

The  WSARA  identifies  10  different  approaches  that  may  be  incorporated  into  an 
MDAP  acquisition  strategy  to  ensure  competition  be  used  if  cost  effective  (WSARA,  2009,  § 
202).  The  list  includes  competitive  prototyping;  dual-sourcing;  unbundling  of  contracts;  use 
of  modular,  open  architecture  to  enable  competition  for  upgrades;  use  of  build-to-print 
approaches;  and  acquisition  of  complete  TDPs — along  with  several  other  approaches. 

These  suggested  measures  involve  competition  among  prime  contractors  and  also  among 
subcontractors  at  such  tiers  as  appropriate.  The  WSARA  views  competition  as  extending 
into  operations  and  sustainment  of  MDAPs. 

The  WSARA  of 2009  Summary 

There  is  no  doubt  that  the  demands  made  in  WSARA  have  increased  the  rigor  and 
discipline  required  in  acquisition  and  will  be  reflected  in  more  careful  cost  estimation, 
increased  caution  in  reviewing  technological  maturity  before  advancing  programs  to  the  next 
acquisition  step  or  phase,  better  systems  engineering  and  test  planning,  and  renewed 
reliance  on  competition.  All  of  these  facets  have  the  potential  to  better  control  LCC. 
Conversely,  all  the  same  facets  introduce  the  potential  for  added  bureaucracy  and 
unnecessary  delay.  The  WSARA  initiatives  address  past  shortcomings  in  MDAP  acquisitions 
that  have  contributed  to  the  increase  of  LCC.  Whether  these  initiatives  will  reduce  cost 
through  better  management  or  increase  cost  through  additional  bureaucracy  remains  to  be 
seen. 


Many  other  facets  of  WSARA  are  described  in  our  201 1  paper,  Total  Ownership 
Cost — Tools  and  Discipline  (Naegle  &  Boudreau,  201 1 ). 

National  Defense  Authorization  Act  for  Fiscal  Year  2010,  Section  805 

The  National  Defense  Authorization  Act  for  FY2010  has  special  relevance  to  life 
cycle  cost,  as  will  be  explained.  In  this  law  Congress  mandated  Product  Support  Manager 
(PSM)  participation  in  MDAPs.  The  law  emphasized  that  the  PSM  works  for  the  PM,  but  is 
also  specifically  tasked  to  focus  on  product  sustainment  (O&S)  cost.  The  PSM  is  tasked  to 
balance  PBL  support  for  optimization.  He  or  she  must  review  and  revalidate  product  support 
strategies  prior  to  a  change  in  strategy  or  every  five  years  (National  Defense  Authorization 
Act,  2010,  §  805).  The  congressional  conferees  recognized  that  product  support 
encompasses  a  wide  range  of  logistics  functions,  including  readiness,  reliability,  availability, 
logistics  burden  (footprint)  reduction — all  of  which  explicitly  or  implicitly  impact  ownership 
cost  (Kobren,  2010,  p.  192).  The  National  Defense  Authorization  Act  for  FY2010  very 
apparently  established  a  position  within  the  MDAP  PM  office  that  is  responsible  for 
sustainment  cost,  to  include  reliability,  which  directly  influences  sustainment  cost. 
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Duncan  Hunter  National  Defense  Authorization  Act  for  Fiscal  Year  2009,  Section 
814,  Configuration  Steering  Boards  for  Cost  Control  Under  Major  Defense  Acquisition 
Programs 

This  law  introduced  a  strong  bias  toward  limiting  design  changes  to  systems.  Note 
that  the  Service  user  representative  is  not  named  as  a  member  of  the  CSB.  The 
presumption  may  be  that  the  user  would  tend  to  encourage  requirements  growth  and  costly 
changes.  The  CSB,  for  its  part,  will  listen  to  the  proposed  change  and  make  the  board 
recommendations  to  the  program  MDA.  In  Part  2,  the  PM  is  directed  to  propose  de-scoping 
options  to  reduce  cost  and  requirements.  The  MDA  is  required  to  coordinate  changes  with 
the  Joint  Staff  and  component  requirements  officials  (i.e.,  user  representatives).  The 
wording  clearly  indicates  a  bias  against  changes  that  will  increase  cost,  or  at  the  least 
deferring  such  changes  to  a  future  block  or  increment. 

Relevant  Studies  and  Reports 

GAO/T-NSI AD-98-1 23  and  Other  GAO  Reports  on  Knowledge  Point  Management 

Knowledge  point  management  can  be  used  to  avoid  program  delays  and  the 
additional  cost  that  accompanies  schedule  delays.  For  more  than  12  years  the  GAO  has 
advocated  the  use  of  knowledge  point  management  to  guide  development  of  warfighting 
systems  and  to  control  the  advancement  of  programs  until  said  systems  have  demonstrated 
their  readiness  to  proceed  to  the  next  step  in  the  development  process  ( Defense 
Acquisition:  Improved  Program  Outcomes,  1998).  The  three  knowledge  points 
recommended  by  the  GAO  are  described  in  the  following  paragraphs. 

Knowledge  Point  1  occurs  near  Milestone  B.  The  user’s  requirements  must  be 
synchronized  with  technology  that  is  mature  enough  to  support  the  endeavor,  allow 
sufficient  time  scheduled  to  succeed,  and  provide  sufficient  funding  to  complete  the 
development  (GAO,  2003,  p.  16).  This  knowledge  point  became  relatively  better  understood 
when  the  Technology  Readiness  Level  Deskbook  was  published  in  2005  (Deputy  Under 
Secretary  of  Defense  for  Science  and  Technology  [DUSD(S&T)j,  2005).  Matching 
requirements  against  resources  is  a  matter  of  discipline  and  having  the  requisite  knowledge 
before  proceeding  is  necessary  because  if  any  one  of  the  several  elements  is  absent  (such 
as  the  application  of  required  technologies  while  they  are  still  immature),  the  program  will 
likely  be  delayed  and  the  impact  on  cost  may  be  severe.  Continuing  GAO  reviews  have 
shown  that  Knowledge  Point  1  demands  enormous  discipline  that  has,  unfortunately,  often 
been  beyond  the  discipline  demonstrated  by  DoD  leadership  over  many  years. 

Knowledge  Point  2  occurs  when  the  design  demonstrates  that  it  is  able  to  meet 
performance  requirements.  The  design  must  be  stable  (i.e.,  90%  of  the  engineering 
drawings  must  be  complete)  and  testing  must  show  that  the  system  performs  at  an 
acceptable  level  (GAO,  2003,  p.  16).  This  point  is  verified  at  the  post-CDR  assessment. 

Knowledge  Point  3  occurs  when  the  system  can  be  manufactured  within  cost, 
schedule,  and  quality  targets  and  operates  reliably  (GAO,  2003,  p.  16).  In  statistical  process 
control  terms,  critical  manufacturing  processes  are  in  control  and  consistently  producing 
within  quality  standards  and  design  tolerances. 

Knowledge  point  management  is  not  new,  but  has  been  an  industry  practice.  The 
same  technique  can,  and  should,  be  applied  to  DoD  system  acquisition. 

Evolutionary  Acquisition 

The  use  of  evolutionary  acquisition  fits  conveniently  with  Knowledge  Point  1 , 
discussed  previously.  Sometimes  technology  does  not  become  mature  as  soon  as  hoped. 
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Depending  on  the  circumstances,  technological  immaturity  might  delay  a  Milestone  B 
decision  and  the  associated  program  new-start.  In  some  cases,  a  technology  that  matures 
more  slowly  than  needed  may  be  substituted  by  an  alternative  technology  that  is  mature  and 
immediately  available.  Plainly,  this  decision  hinges  on  whether  or  not  the  developing  system 
can  result  in  an  increment  of  useful  warfighting  capability — as  determined  by  the 
sponsor/user.  Even  when  this  happens,  the  program  faces  a  difficult  path  that  requires 
“extra”  milestones  that  are  exhausting  to  program  office  staff.  Such  is  the  nature  of 
evolutionary  acquisition — avoiding  one  dilemma  and  replacing  it  with  another.  The 
evolutionary  approach  places  heavy  demands  on  a  program  office,  which  must  prepare  for  a 
series  of  otherwise  unnecessary  milestones.  Is  it  worth  it? 

The  logistics  impact  of  evolutionary  acquisition  cannot  be  ignored,  either.  A  result  of 
evolutionary  acquisition  will  either  be  multiple  configurations  or  expensive 
modification/upgrades.  Such  cost  impacts  might  play  out  for  many  years  or  even  for  the 
lifetime  of  the  warfighting  system.  This  may  be  associated  training  issues,  repair  parts 
configuration  issues,  software  patches,  and  operational  impacts.  The  cost  of  evolutionary 
acquisition  could  conceivably  approach  or  even  exceed  the  original  cost  of  the  program 
delay. 

The  right  answer  in  acquisition  depends  on  the  circumstances.  The  effect  on 
ownership  cost  should  always  be  one  of  the  metrics  used  to  select  the  best  course  of  action. 

GAO  Report  10-717 

In  July  2010,  the  GAO  (2010)  published  Defense  Management:  DOD  Needs  Better 
Information  and  Guidance  to  More  Effectively  Manage  and  Reduce  Operating  and  Support 
Costs  of  Major  Weapon  Systems  (GAO  10-717).  This  report  painted  a  dreary  picture  of 
relevant  cost  databases.  The  GAO  found  that  important  O&S  cost-estimate  documents  for 
aviation  systems  had  not  been  retained  and  that  there  were  apparent  gaps  in  the  DoD’s 
ability  to  capture  actual  O&S  costs  through  the  Services’  Visibility  and  Maintenance  of 
Operations  and  Support  Costs  (VAMOSC)  databases  (GAO,  2010,  p.  16).  Data  in  VAMOSC 
and  other  Service  information  systems  or  sources  was  inaccurate  and  incomplete  (GAO, 
2010,  pp.  16-20).  The  report  stated  that  the  important  MDAP  system  life  cycle  cost 
estimates  were  not  being  routinely  retained  or  updated,  nor  was  there  policy  requiring  that 
this  be  done.  The  GAO  pointed  out  that  there  were  no  agreed-to  O&S  cost  elements  or 
metrics  for  tracking  and  assessing  actual  O&S  cost  performance  for  the  various  categories 
of  weapon  systems.  Additionally,  operational  costs  also  were  affected  by  unexpected 
changes  in  OPTEMPO  (specifically,  flying  hours;  GAO,  2010,  p.  22).  Although  both  those 
factors  might  upset  budget  predictions,  they  need  not  upset  performance  predictions;  rather, 
if  shown  as  “cost  per  usage,”  reasonable  comparisons  might  show  the  weapon  system’s 
performance  against  baseline  performance.  Cost  per  mile  or  cost  per  flying  hour  or  round 
fired  could  be  compared  to  early  cost  estimates,  as-tested  costs,  and  changes  in  cost  per 
year.  Such  comparisons  would  never  be  perfect,  but  they  would  suggest  whether  a  weapon 
system  was  performing  within  the  expected  range. 

Looking  specifically  at  aviation  systems  across  the  Services,  the  GAO  reported  that 
most  systems  had  no  record  of  O&S  cost  estimates  related  to  key  milestone  decisions.  Two 
aircraft  systems,  the  Air  Force  F-22A  fighter  and  the  Navy  F-A  18F/G,  did  have  some 
recorded  O&S  cost  estimates  (GAO,  2010,  pp.  24-26).  The  two  cited  examples  suggest  the 
seriousness  of  O&S  cost-estimating  inaccuracy  and/or  cost  growth.  F-22A  actual  cost  per 
flight  hour  in  2007  was  $55,783 — 67%  higher  than  the  $33,762  that  had  been  projected  in 
the  2007  President’s  Budget.  Similarly,  on  a  flight-hour  basis,  the  Navy  F-A  18E/F  cost 
$15,346  per  flight  hour  of  operation — 40%  higher  than  the  $10,979  predicted  in  1999. 
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Institute  for  Defense  Analyses  Study:  The  Major  Causes  of  Cost  Growth  in  Defense 
Acquisition 

The  2009  Institute  for  Defense  Analyses  (IDA)  study,  led  by  Gene  Porter,  examined 
1 1  MDAP  systems  that  had  exhibited  significant  cost  growth  between  1995  and  2006.  The 
primary  causes  of  cost  growth  stemmed  from  two  defects:  “weaknesses  in  management 
visibility,  direction,  and  oversight”  and  “weaknesses  in  initial  program  definition  and  costing,” 
neither  of  which  was  a  new  phenomenon  (Porter  et  al.,  2009,  pp.  ES-6-ES-14).  Much  of  the 
blame  for  the  first  weakness  was  “a  general  lack  of  discipline”  (Porter  et  al.,  2009,  p.  ES-6). 

Porter  et  al.  (2009)  make  a  series  of  recommendations  that  are  intended  to  address 
the  causes  of  cost  growth  reflected  in  their  study;  their  recommendations  are  supportive  of 
the  goals  of  the  WSARA  of  2009  (Porter  et  al.,  2009,  pp.  ES-1 5-ES-1 8). 

DOT&E  Initiative  on  Reliability  Growth 

In  his  memorandum,  State  of  Reliability,  J.  Michael  Gilmore,  the  Director  of 
Operational  Test  and  Evaluation  (DOT&E,  2010),  made  the  link  that  poor  reliability  is  a 
major  contributor  to  LCC.  The  implication  is  that  the  long-held  28-72  LCC  statistics  could  be 
altered  by  front-end  attention  to  reliability  growth.  That  is,  investing  more  RDTE  funding  in 
reliability  improvement  at  the  front  end  could  result  in  higher  reliability  components  that 
would  cost  less  to  operate,  malfunctioning  less  often.  The  remarkable  thing  here  is  that 
program  leadership  has  tried  to  improve  reliability  in  many,  if  not  all,  programs.  Gilmore 
made  reference  to  a  recently  published  reliability  standard,  ANSI/GEIA-STD-0009,  which 
should  be  employed. 

Policy  Pronouncements 

The  OSD  implemented  the  2009  version  of  the  WSARA  on  December  4,  2010, 
through  the  USD(AT&L)  publication  of  Directive  Type  Memorandum  (DTM)  09-027 
(USD[AT&L],  2009).  About  10  months  later,  on  October  21, 2010,  the  USD(AT&L)  amended 
the  original  document,  establishing  a  date  by  which  the  DoDI  5000.02  had  to  be  revised 
(USD[AT&L],  2010a). 

Target  Affordability  and  Control  Cost  Growth  for  AC  AT  I  Programs 

Corollary  to  WSARA  implementation,  the  USD(AT&L)  published  the  Implementation 
Directive  for  Better  Buying  Power— Obtaining  Greater  Efficiency  and  Productivity  in  Defense 
Spending  (USD[AT&L],  2010b).  The  intent  of  this  implementation  directive  was  to  reach 
beyond  WSARA  mandates  to  obtain  greater  affordability-based  decision-making  in 
warfighting  system  programs.  Specifically,  its  goal  was  to  mandate  affordability  as  a 
requirement.  PMs  are  now  required  to  treat  affordability  as  a  key  performance  parameter 
(KPP)  at  Milestone  A.  The  affordability  target  is  to  be  stated  in  two  metrics:  average  unit 
acquisition  cost  and  average  annual  operating  and  support  cost  per  unit.  These  metrics  will 
be  the  basis  for  pre-Milestone  B  decision-making  and  systems  engineering  trade-off 
analysis  to  establish  cost  and  schedule  trade  space.  Such  a  mandate  requires  a  database 
similar  to  the  one  Roper  described  (2010,  pp.  71-73). 

There  have  been  significant  other  recent  directive-type  memoranda  (DTMs)  that 
affect  ownership  cost  and  affordability.  Some  of  these  DTMs  are  discussed  in  more  detail  in 
our  201 1  paper,  Total  Ownership  Cost — Tools  and  Discipline  (Naegle  &  Boudreau,  2011). 

A  Specific  Navy  Initiative:  Gate  Reviews 

The  Navy  has  instituted  a  series  of  reviews,  termed  “gate  reviews,”  to  better  control 
program  development  cost.  The  Navy  Total  Ownership  Cost  Guidebook  (Department  of  the 
Navy  [DoN],  2010;  published  concurrently  with  SECNAVINST  5000. 2E)  depicts  a  series  of 
10  gate  reviews  that  stretch  across  the  pre-acquisition  and  acquisition  phases  and  into  the 
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sustainment  phase.  Each  gate  review  asks  tailored  cost  questions  relevant  to  the  specific 
life  cycle  event  (DoN,  2010,  pp.  4-32).  The  complete  array  of  gate  reviews  is  as  follows: 

•  Gate  1 — Initial  Capabilities  Document 

•  Gate  2 — Analysis  of  Alternatives 

•  Gate  3 — Capability  Development  Document 

•  Gate  4 — System  Design  Specification 

•  Gate  5 — RFP  for  Engineering  and  Manufacturing  Development  Contract 

•  Gate  6  Reviews — specifically,  Integrated  Baseline  Review,  Post  Critical 
Design  Review,  Capability  Production  Document,  Pre-Full  Rate  Production 
Decision  Review,  and  Sustainment  Sufficiency  Review(s) 

At  each  gate  review,  formal  design  review,  and  assessment,  programs  must 
demonstrate  progress  toward  their  affordability  initiatives,  with  strong  consideration  in 
mitigation  or  reduction  of  TOC.  The  Navy’s  intent  is  to  change  the  culture  from  what  the 
authors  of  this  working  paper  perceive  as  a  shortsighted  goal  of  obtaining  funds  for 
development  and  procurement  to  the  more  complete  perspective  of  total  life  cycle  cost 
affordability. 

Gate  Review  1 ,  which  is  intended  to  shape  the  analysis  of  alternatives  (AoA  )  study 
analysis,  requires  consideration  of  O&S  costs  based  on  current  or  similar  systems.  AoA 
study  TOC  guidance  is  intended  to  be  sufficiently  detailed  to  inform  and  support  the 
selection  of  a  materiel  solution  from  among  the  various  AoA  candidates. 

Intermediate  gate  reviews  are  coupled  to  existing  systems  engineering  and 
acquisition  milestone  review  points.  These  reviews  become  a  forum  to  assess  whether 
program  trade-offs  and  decisions  are  controlling  life  cycle  cost  and  whether  the  program  is 
continuing  on  the  correct  affordability  azimuth.  Each  of  the  gate  reviews  requires  briefing  of 
specific  cost  charts,  making  it  unlikely  that  cost  growth  and  schedule  slippage  can  be 
obscured. 

The  Gate  6  Sustainment  Review(s),  accomplished  post-IOC,  examine  the  warfighting 
system’s  actual  performance  data  compared  to  the  system’s  KPP  thresholds  and  the 
warfighting  system’s  actual  life  cycle  cost  compared  to  its  prior  estimates  of  ownership  cost. 

In  the  aggregate,  Gate  Reviews  provide  for  oversight  and  governance  of  MDAP 
system  developments.  In  a  wider  sense,  Gate  Reviews  provide  a  forum  for  lessons  learned 
regarding  TOC  while  controlling  the  affordability  of  individual  systems — and,  hence,  the 
broader  portfolios  of  warfighting  systems — throughout  the  developmental,  production,  and 
sustainment  phases  of  warfighting  systems. 

Other  Initiatives 

Controls  on  Software  Development 

Driving  the  Software  Requirements  and  Architectures  for  System 

Supportability 

While  the  tools  and  techniques  described  in  this  section  were  designed  for  the 
software  components,  they  would  be  just  as  effective  for  any  non-software  component  as 
they  are  systems  engineering  (SE)  oriented.  The  systems  engineering  process  (SEP)  focus 
used  does  not  attempt  to  separate  software  from  other  components,  so  all  system 
components  would  benefit  from  using  these  tools  and  techniques. 
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Software  Supportability  Analysis 

As  with  hardware  system  components,  software  supportability  attributes  must  be 
designed  into  the  system  architecture.  Many  hardware-oriented  engineering  fields  are  now 
quite  mature,  so  that  a  number  of  supportability  attributes  would  be  automatically  included  in 
any  competent  design,  even  if  they  were  not  specified  by  the  user  community.  For  example, 
the  state  of  maturity  for  the  automotive  engineering  field  means  that,  in  any  automotive- 
related  program,  there  would  be  supportability  designs  allowing  for  routine  maintenance  of 
system  filters,  lubricants,  tires,  brakes,  batteries,  and  other  normal  wear-out  items.  There  are 
few,  if  any,  corresponding  supportability  design  attributes  that  would  be  automatically 
included  in  even  the  best  software  construct.  Virtually  all  of  the  software  supportability 
attributes  required  must  be  explicitly  specified  because  they  would  not  likely  be  included  in 
the  design  architecture  without  clearly  stated  requirements.  With  software,  you  get  what  you 
specify  and  very  little  else.  So  how  does  one  ensure  that  required  software  supportability 
attributes  are  not  overlooked? 

Logistics  Supportability  Analysis  (LSA),  performed  extremely  early,  is  one  of  the  keys 
for  developing  the  system  supportability  attributes  needed  and  expected  by  the  warfighter. 
The  F/A  18  Super  Hornet  aircraft  was  designed  for  higher  reliability  and  improved  ease  of 
maintenance  compared  to  its  predecessors  (“F/A  18,”  2011)  because  of  warfighter  needs  for 
generating  combat  power  in  the  form  of  available  aircraft  sorties.  The  LSA  performed  on  the 
F/A  18  determined  that  a  design  fostering  higher  reliability  and  faster  maintenance 
turnaround  time  (the  engines  are  attached  to  the  airframe  at  10  locations  and  can  be 
changed  in  about  20  minutes  by  a  four-man  team)  would  result  in  more  aircraft  being 
available  to  the  commander  when  needed.  The  concept  for  software  LSA  is  no  different,  but 
implementing  sound  supportability  analyses  on  the  software  components  has  been,  at  best, 
spotty  and,  at  worst,  completely  lacking. 

To  assist  in  effective  software  LSA,  a  focus  on  the  following  elements  is  key: 
Maintainability,  Upgradeability,  Interoperability/Interfaces,  Reliability,  and  Safety  & 

Security — MUIRS. 

Maintainability 

The  amount  of  elapsed  time  between  initial  fielding  and  the  first  required  software 
maintenance  action  can  probably  be  measured  in  hours,  not  days.  The  effectiveness  and 
efficiency  of  these  required  maintenance  actions  is  dependent  on  several  factors,  but  the 
software  architecture  that  was  developed  from  the  performance  specifications  provided  is 
critical.  The  DoD  must  influence  the  software  architecture  through  the  performance 
specification  process  to  minimize  the  cost  and  time  required  to  perform  essential 
maintenance  tasks. 

Maintenance  is  one  area  in  which  software  is  fundamentally  different  from  hardware. 
Software  is  one  of  the  very  few  components  in  which  we  know  that  the  fielded  product  has 
shortcomings,  and  we  field  it  anyway.  There  are  a  number  of  reasons  why  this  happens;  for 
instance,  there  is  typically  not  enough  time,  funding,  or  resources  to  find  and  correct  every 
error,  glitch,  or  bug,  and  not  all  of  these  are  worth  the  effort  of  correcting.  Knowing  this, 
there  must  be  a  sound  plan  and  resources  immediately  available  to  quickly  correct  those 
shortcomings  that  do  surface  during  testing  and  especially  those  that  arise  during 
warfighting  operations.  Even  when  the  system  software  is  operating  well,  changes  and 
upgrades  in  other  interfaced  hardware  and  software  systems  will  drive  some  sort  of  software 
maintenance  action  to  the  system  software.  In  other  words,  there  will  be  a  continuous  need 
for  software  maintenance  in  the  planned  complex  SoS  architecture  envisioned  for  net- 
centric  warfare. 
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Because  the  frequency  of  required  software  maintenance  actions  is  going  to  be 
much  higher  than  in  other  systems,  the  cost  to  perform  these  tasks  is  likely  to  be  higher  as 
well.  One  of  the  reasons  for  this  is  that  software  is  not  maintained  by  “maintainers,”  as  are 
most  hardware  systems,  but  is  maintained  by  the  same  type  of  people  that  originally 
developed  it — software  engineers.  These  engineers  will  be  needed  immediately  upon 
fielding,  and  a  number  will  be  needed  throughout  the  lifespan  of  the  system  to  perform 
maintenance,  add  capabilities,  and  upgrade  the  system.  There  are  several  models  available 
to  estimate  the  number  of  software  engineers  that  will  be  needed  for  support;  planning  for 
funding  these  resources  must  begin  very  early  in  the  process.  Because  the  DoD  has  a  very 
limited  capability  for  supporting  software  internally,  early  software  support  is  typically 
provided  by  the  original  developer  and  is  included  in  the  RFP  and  proposal  for  inclusion  into 
the  contract  or  as  a  follow-on  Contractor  Logistics  Support  (CLS)  contract. 

Upgradeability 

A  net-centric  environment  composed  of  numerous  systems  developed  in  an 
evolutionary  acquisition  model  will  create  an  environment  of  almost  continuous  change  as 
each  system  upgrades  its  capabilities  over  time.  System  software  will  have  to  accommodate 
the  changes  and  will  have  to,  in  turn,  be  upgraded  to  leverage  the  consistently  added 
capabilities.  The  software  architecture  design  will  play  a  major  role  in  how  effective  and 
efficient  capabilities  upgrades  are  implemented,  so  communicating  the  known,  anticipated, 
and  likely  system  upgrades  will  impact  how  the  software  developer  designs  the  software  for 
known  and  unknown  upgrades. 

Trying  to  anticipate  upgrade  requirements  for  long-lived  systems  is  extremely 
challenging  to  materiel  developers,  but  is  well  worth  their  effort.  Unanticipated  software 
changes  in  the  operational  support  phase  cost  50-200  times  the  cost  in  early  design,  so  any 
software  designed  to  accommodate  an  upgrade  that  is  never  realized  costs  virtually  nothing 
when  compared  to  changing  software  later  for  a  capability  that  could  have  been  anticipated. 
For  example,  the  Army  Tactical  Missile  System  (ATACMS)  Unitary  was  a  requirement  to 
modify  the  missile  from  warhead  air  delivery  to  surface  detonation — that  is,  flying  the 
warhead  to  the  ground.  The  contract  award  for  the  modification  was  $119  million.  The 
warhead  was  not  new  technology,  nor  particularly  challenging  to  integrate  with  the  missile 
body.  The  vast  majority  of  this  cost  was  to  reengineer  the  software  to  guide  the  missile  to 
the  surface.  Had  there  been  an  upgrade  requirement  for  this  type  of  mission  in  the  original 
performance  specification,  this  original  cost  (including  potential  upgrades,  even  if  there  were 
10  other  upgrade  requirements  that  were  never  applied)  would  have  been  a  fraction  of  this 
modification  cost. 

Interfaces/Interoperability 

OA  design  focuses  on  the  strict  control  of  interfaces  to  ensure  the  maximum  flexibility 
in  adding  or  changing  system  modules,  whether  they  are  hardware  or  software  in  nature. 
This  presupposes  that  the  system  modules  are  known — which  seems  logical,  as  most 
hardware  modules  are  well-defined  and  bounded  by  both  physics  and  mature  engineering 
standards.  In  sharp  contrast  to  hardware,  software  modularity  is  not  bounded  by  physics, 
and  there  are  very  few  software  industry  standards  for  the  modular  architecture  in  software 
components.  This  is  yet  another  area  in  which  the  software  developer  needs  much  more 
information  about  operational,  maintenance,  reliability,  safety,  and  security  performance 
requirements,  as  well  as  current,  planned,  and  potential  system  upgrades.  These 
requirements,  once  well  defined  and  clearly  communicated,  will  drive  the  developer  to 
design  a  software  modular  architecture  supporting  OA  performance  goals.  For  example,  if  a 
system  uses  a  Global  Positioning  System  (GPS)  signal,  it  is  likely  that  the  GPS  will  change 
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over  the  life  of  the  system.  Knowing  this,  the  software  developer  creates  a  corresponding 
discrete  software  module  that  is  much  easier  and  less  expensive  to  interface  with,  change, 
and  upgrade  along  with  the  GPS  system. 

With  the  system  software  modular  architecture  developed,  the  focus  returns  to  the 
interfaces  between  hardware  and  software  modules,  as  well  as  to  the  external  interfaces 
needed  for  the  desired  interoperability  of  the  net-centric  force.  Software  is,  of  course,  one  of 
the  essential  enablers  for  interoperability  and  provides  a  powerful  tool  for  interfacing 
systems,  including  systems  that  were  not  designed  to  work  together.  Software  performing 
the  function  of  “middleware”  allows  legacy  and  other  dissimilar  systems  to  interoperate. 
Obviously,  this  interoperation  provides  a  significant  advantage,  but  it  comes  with  a  cost  in 
the  form  of  maintainability,  resources,  and  system  complexity.  As  software  interfaces  with 
other  components  and  actually  performs  the  interface  function,  controlling  it  and  ensuring 
the  interfaces  provide  the  desired  OA  capability  become  major  software-management  and 
software-discipline  challenges. 

One  method  being  employed  by  the  DoD  attempts  to  control  the  critical  interfaces 
through  a  set  of  parameters  or  protocols  rather  than  through  active  management  of  the 
network  and  network  environment.  This  method  falls  short  on  several  levels.  It  fails  to 
understand  and  control  the  effects  of  aggregating  all  of  the  systems  in  a  net-centric  scheme. 
For  instance,  each  individual  system  may  meet  all  protocols  for  bandwidth,  but  when  all 
systems  are  engaged  on  the  network,  all  bandwidth  requirements  are  aggregated  on  the 
network — overloading  the  total  bandwidth  available  for  all  systems.  In  addition,  members  of 
the  Software  Engineering  Institute  (SEI)  noted, 

While  these  standards  may  present  a  step  in  the  right  direction,  they  are 
limited  in  the  extent  to  which  they  facilitate  interoperability.  At  best,  they 
define  a  minimal  infrastructure  that  consists  of  products  and  other  standards 
on  which  systems  can  be  based.  They  do  not  define  the  common  message 
semantics,  operational  protocols,  and  system  execution  scenarios  that  are 
needed  for  interoperation.  They  should  not  be  considered  system 
architectures.  For  example,  the  C4ISR  domain-specific  information  (within  the 
JTA)  identifies  acceptable  standards  for  fiber  channels  and  radio 
transmission  interfaces,  but  does  not  specify  the  common  semantics  of 
messages  to  be  communicated  between  C4ISR  systems,  nor  does  it  define 
an  architecture  for  a  specific  C4ISR  system  or  set  of  systems.  (Morris  et  al., 
2004,  p.  38) 

Clearly,  understanding  and  controlling  the  interfaces  is  critical  for  effective 
interoperation  at  both  the  system  and  SoS  levels.  The  individual  PM  must  actively  manage 
all  systems’  interfaces  impacting  OA  performance,  and  a  network  PM  must  do  the  same  for 
the  critical  network  interfaces.  Due  to  this  necessity  of  constant  management,  a  parameters- 
and-protocols  approach  to  net-centric  OA  performance  is  unlikely  to  produce  the  capabilities 
and  functionality  expected  by  the  warfighter. 

Understanding  the  software  interfaces  begins  with  the  software  architecture; 
controlling  the  interfaces  is  a  unique  challenge  encompassing  the  need  to  integrate  legacy 
and  dissimilar  systems  and  the  lack  of  software  interface  standards  within  the  existing 
software  engineering  environment.  As  stated  earlier,  the  architecture  needs  to  be  driven 
through  detailed  performance  specifications,  which  will  help  define  the  interfaces  to  be 
controlled.  An  effective  method  for  controlling  the  interfaces  is  to  intensely  manage  a  well- 
defined  Interface  Control  Document  (ICD),  which  should  be  a  Contract  Data  Requirements 
List  (CDRL)  deliverable  on  any  software-intensive  or  networked  system. 
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Reliability 

While  the  need  for  highly  reliable  weapon  systems  is  obvious,  the  impact  on  total 
system  reliability  of  integrating  complex  software  components  is  not  so  obvious.  Typically, 
as  system  complexity  increases,  maintaining  system  reliability  becomes  more  of  a 
challenge.  Add  the  complexity  of  effectively  networking  an  SoS  (all  of  which  are  individually 
complex)  to  a  critical  warfighting  capability  that  is  constantly  evolving  over  time,  and 
reliability  becomes  daunting. 

Once  again,  the  software  developer  must  have  an  understanding  of  reliability 
requirements  before  crafting  the  software  architecture  and  developing  the  software 
applications.  Highly  reliable  systems  often  require  redundant  capability,  and  this  holds  true 
for  software  components  as  well.  In  addition,  software  problems  tend  to  propagate,  resulting 
in  a  degradation  of  system  reliability  over  time.  For  example,  a  Malaysian  Airlines  Boeing 
777  suffered  several  flight  control  problems,  resulting  in  the  following:  a  near  stall  situation, 
contradicting  instrument  indications,  false  warnings,  and  difficulty  controlling  the  aircraft  in 
both  autopilot  and  manual  flight  modes.  The  problems  were  traced  to  software  in  an  air  data 
inertial  reference  unit  that  was  feeding  erroneous  data  to  the  aircraft’s  primary  flight 
computer  (PFC),  which  is  used  in  both  autopilot  and  manual  flight  modes.  The  PFC 
continued  to  try  to  correct  for  the  erroneous  data  received,  adjusting  flight  control  surfaces  in 
all  modes  of  flight,  displaying  indications  that  the  aircraft  was  approaching  stall  speed  and 
overspeed  limits  simultaneously,  and  causing  wind  shear  alarms  to  sound  close  to  landing 
(Dornheim,  2005,  p.  46).  It  is  critical  for  system  reliability  that  the  software  developers 
understand  how  outputs  from  software  applications  are  used  by  interfaced  systems  so  that 
appropriate  reliability  safeguards  can  be  engineered  into  the  developed  software. 

Software  that  freezes  or  shuts  down  the  system  when  an  anomaly  occurs  is  certainly 
not  reliable  nor  acceptable  for  critical  weapon  systems;  yet,  these  characteristics  are 
prevalent  in  commercially  based  software  systems.  Mission  reliability  is  a  function  of  the 
aggregation  of  the  system’s  subcomponent  reliability,  so  every  software  subcomponent  is 
contributing  to  or  detracting  from  that  reliability.  The  complexity  of  software  makes 
understanding  all  failure  modes  nearly  impossible,  but  there  are  many  techniques  that 
software  developers  can  employ  when  designing  the  architecture  and  engineering  the 
applications  to  improve  the  software  component  reliability.  Once  requirements  are  clearly 
communicated  to  the  developers,  the  software  can  be  engineered  with  redundancy  or  “safe 
mode”  capabilities  to  vastly  improve  mission  reliability  when  anomalies  occur.  The  key  is 
identifying  the  reliability  requirements  and  making  them  clear  to  the  software  developers. 

Safety  &  Security 

Very  few  software  applications  have  the  required  safety  margins  associated  with 
critical  weapon  systems  used  by  warfighters  in  combat  situations — where  they  are 
depending  on  these  margins  for  their  survival.  Typically,  the  software  developers  have  only 
a  vague  idea  of  what  their  software  is  doing  and  how  critical  that  function  is  to  the  warfighter 
employing  the  weapon  system.  Safety  performance  must  be  communicated  to  the  software 
developers  from  the  beginning  of  development  so  they  have  the  link  between  software 
functionality  and  systems  safety.  For  example,  suppose  a  smart  munition  senses  that  it  does 
not  have  control  of  a  critical  directional  component,  and  it  calculates  that  it  cannot  hit  the 
intended  target.  The  next  set  of  instructions  the  software  provides  to  the  malfunctioning 
system  may  well  be  critical  to  the  safety  of  friendly  troops,  so  software  developers  must 
have  the  necessary  understanding  of  operational  safety  to  decide  how  to  code  the  software 
for  what  will  happen  next. 
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Software  safety  is  clearly  linked  with  reliability  since  software  that  is  more  reliable  is 
inherently  safer.  It  is  critical  that  the  software  developer  understands  how  the  warfighter 
expects  the  software  to  operate  in  abnormal  situations,  in  degraded  modes,  and  when 
inputs  are  outside  of  expected  values.  Much  commercially  based  software  simply  ceases  to 
function  under  these  conditions  or  gives  error  messages  that  supersede  whatever  function 
was  being  performed,  none  of  which  are  acceptable  in  combat  operations. 

With  software  performing  so  many  critical  functions,  there  is  little  doubt  that  software 
applications  are  a  prime  target  for  anyone  opposing  U.S.  and  Allied  forces.  Critical  weapon 
system  and  networking  software  must  be  resistant  to  hacking,  spoofing,  mimicking,  and  all 
other  manner  of  attack.  There  must  be  capabilities  for  isolating  attacks  and  portions  of 
networks  that  have  been  compromised  without  losing  the  ability  to  continue  operations  in 
critical  combat  situations.  The  software  developer  must  know  that  all  of  these  capabilities 
are  essential  before  he  or  she  constructs  software  architectures  and  software  programs,  as 
this  knowledge  will  be  very  influential  for  the  software  design  and  application  development. 
The  SEI’s  Quality  Attribute  Workshop  states,  “As  an  example,  consider  security.  It  is  difficult, 
maybe  even  impossible,  to  add  effective  security  to  a  system  as  an  afterthought. 

Component  as  well  as  communication  mechanisms  and  paths  must  be  designed  or  selected 
early  in  the  lifecycle  to  satisfy  security  requirements”  (Barbacci  et  al.,  2003,  p.  2). 

Interoperability  challenges  are  increased  when  the  SoS  has  the  type  of  security 
requirements  needed  by  the  DoD.  Legacy  systems  and  existing  security  protocols  will  likely 
need  to  be  considered  before  other  security  architecture  can  be  effectively  designed.  OA 
capabilities  will  be  hampered  by  the  critical  need  for  security;  both  must  be  carefully 
balanced  to  optimize  system  performance  and  security.  This  balance  of  OA  and  security 
must  be  managed  by  the  DoD  and  not  the  software  developer. 

Physical  security  schemes  and  operating  procedures  will  also  have  an  impact  on  the 
software  architecture.  For  example,  many  communication  security  (COMSEC)  devices  need 
only  routine  security  until  the  keys,  usually  software  programs,  are  applied;  then,  much  more 
stringent  security  procedures  are  implemented.  Knowledge  of  this  security  feature  would  be 
a  key  requirement  of  the  developer;  he  or  she  must  understand  how  and  when  the  critical 
software  pieces  are  uploaded  to  the  COMSEC  device.  The  same  holds  true  for  weapon 
systems  that  upload  sensitive  mission  data  just  prior  to  launch. 

Residual  software  on  equipment  or  munitions  that  could  fall  into  enemy  hands 
presents  another  type  of  security  challenge  that  needs  to  be  addressed  during  application 
development.  For  example,  the  ATACMS  missile  air-delivers  some  of  its  warheads,  leaving 
the  missile  body  to  free  fall  to  the  surface.  It  is  very  conceivable  that  the  body  could  be  intact 
and,  of  course,  unsecured.  If  critical  mission  software  was  still  within  the  body  and  found  by 
enemy  forces,  valuable  information  might  be  gleaned  from  knowing  how  the  system  finds  its 
targets.  The  government  would  certainly  want  the  developer  to  design  the  applications  in  a 
way  that  would  make  anything  recovered  useless  to  the  enemy,  but  this  is  a  capability  that  is 
not  intuitive  to  software  developers  (Naegle,  2006,  pp.  17-25). 

Effective  Software  Development  Tools  Supporting  System  TOC  Analyses 

Software  Engineering  Institute’s  (SEI)  Quality  Attribute  Workshop  (QAW) 

The  QAW  is  designed  to  help  identify  a  complete  (or  as  complete  as  possible) 
inventory  of  system  software  requirements  through  analysis  of  system  quality  attributes.  One 
of  the  intents  is  to  develop  the  derived  and  implied  requirements  from  the  user-stated 
requirements,  which  is  a  necessary  step  when  user-stated  requirements  are  provided  in 
terms  of  capabilities  needed  as  prescribed  by  the  Joint  Capabilities  Integration  Development 
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System  (JCIDS)  process.  A  system’s  TOC,  and  those  elements  that  contribute  to  TOC,  are 
system  quality  attributes.  Although  obviously  important  to  the  warfighter,  the  associated 
operations  and  support,  training/education,  and  facility  costs  are  rarely  addressed  in  much 
detail  and  need  to  be  derived  from  stated  requirements  or  augmented  with  implied 
requirements  through  the  QAW  process,  or  something  similar. 

The  QAW  helps  provide  a  facilitating  framework  and  process  designed  to  more  fully 
develop  the  derived  and  implied  requirements  that  are  critical  to  clearly  communicate  to 
potential  contractors  and  software  developers.  Including  a  robust  LSA  process  using  the 
MUIRS  focus  elements,  described  previously,  within  the  QAW  process  will  likely  significantly 
improve  requirements  analysis  for  those  associated  TOC  elements  and  vastly  improve  the 
accuracy  of  system  TOC  projections.  While  improving  system  requirements  development, 
the  QAW  is  designed  to  work  with  another  SEI  process  called  the  Architectural  Trade-off 
Analysis  MethodologySM  (ATAMsm)  to  further  improve  the  understanding  of  the  system  for 
potential  contractors  and  software  developers. 

SEI’s  Architectural  Trade-Off  Analysis  Methodology SM  (ATAMsm) 

The  SEI’s  ATAMsm  is  an  architectural  analysis  tool  designed  to  evaluate  design 
decisions  based  on  the  quality  attribute  requirements  of  the  system  being  developed.  The 
methodology  is  a  process  for  determining  whether  the  quality  attributes,  including  TOC 
attributes,  are  achievable  by  the  architecture  as  it  has  been  conceived  before  enormous 
resources  have  been  committed  to  that  design.  One  of  the  main  goals  is  to  gain  insight  into 
how  the  quality  attributes  trade  off  against  each  other  (Kazman,  Klein,  &  Clements,  2000,  p. 
1)- 

Within  the  systems  engineering  process  (SEP),  the  ATAMsm  provides  the  critical 
requirements  loop  process,  tracing  each  requirement  or  quality  attribute  to  corresponding 
functions  reflected  in  the  software  architectural  design.  Whether  ATAMsm  or  another 
analysis  technique  is  used,  this  critical  SEP  must  be  performed  to  ensure  that  functional-  or 
object-oriented  designs  meet  all  stated,  derived,  and  implied  warfighter  requirements.  In 
complex  systems  development,  such  as  weapon  systems,  half  or  more  than  half  of  the  total 
software  development  effort  will  be  expended  in  the  architectural  design  process.  Therefore, 
DoD  PMs  must  ensure  that  the  design  is  addressing  requirements  in  context  and  that  the 
resulting  architecture  has  a  high  probability  of  producing  the  warfighters’  JCIDS  stated, 
derived,  or  implied  requirements. 

The  ATAMsm  focuses  on  quality  attribute  requirements,  so  it  is  critical  to  have  precise 
characterizations  for  each.  To  characterize  a  quality  attribute,  the  following  questions  must 
be  answered: 

•  What  are  the  stimuli  to  which  the  architecture  must  respond? 

•  What  is  the  measurable  or  observable  manifestation  of  the  quality  attribute  by 
which  its  achievement  is  judged? 

•  What  are  the  key  architectural  decisions  that  impact  achieving  the  attribute 
requirement?  (2000,  p.  5) 

The  ATAMsm  scenarios  are  a  key  to  providing  the  necessary  information  to  answer 
the  first  two  questions,  driving  the  software  engineer  to  design  the  architecture  to  answer  the 
third.  This  is  a  critical  point  at  which  all  of  the  MUIRS  elements  need  to  be  considered  and 
appropriate  scenarios  developed. 

The  ATAMsm  uses  three  types  of  scenarios:  use-case  scenarios  involve  typical  uses 
of  the  system  to  help  understand  quality  attributes  in  the  operational  context;  growth 
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scenarios  involve  anticipated  design  requirements,  including  upgrades,  added  interfaces 
supporting  SoS  development,  and  other  maturity  needs;  and  exploratory  scenarios  involve 
extreme  conditions  and  system  stressors,  including  Failure  Modes  and  Effects  Criticality 
Analysis  (FMECA)  scenarios  (Kazman  et  al.,  2000,  pp.  13-15).  As  depicted  in  Figure  1,  the 
scenarios  build  on  the  basis  provided  in  the  JCIDS  documents  and  requirements  developed 
through  the  QAW  process.  These  processes  lend  themselves  to  development  in  an 
Integrated  Product  Team  (IPT)  environment  led  by  the  user/combat  developer  and  including 
all  of  the  system’s  stakeholders.  The  IPT  products  will  include  a  set  of  scenarios,  prioritized 
by  the  needs  of  the  warfighter  for  system  capability.  The  prioritization  process  provides  a 
basis  for  architecture  trade-off  analyses.  When  fully  developed  and  prioritized,  the  scenarios 
provide  a  more  complete  understanding  of  requirements  and  quality  attributes  in  context 
with  the  operation  and  support  (including  all  of  the  MUIRS  elements)  of  the  system  over  its 
life  cycle.  A  more  complete  understanding  of  the  system’s  TOC  elements  should  emerge 
from  this  type  of  analysis. 
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Figure  1.  QAW  &  ATAM  Integration  Into  Software  Life  Cycle  Management 

Just  as  the  QAW  process  provides  a  methodology  supporting  RFP,  source-selection 
activities,  and  the  Software  Specification  and  System  Requirements  Reviews  (SSR  and 
SRR),  the  ATAMsm  provides  a  methodology  supporting  design  analyses,  test  program 
activities,  and  the  System  Functional  and  Preliminary  Design  Reviews  (SFR  and  PDR).  The 
QAW  and  ATAMsm  methodologies  are  probably  not  the  only  effective  methods  supporting 
software  development  efforts,  but  they  fit  particularly  well  with  the  DoD’s  goals,  models,  and 
SEP  emphasis.  The  user/combat  developer  (blue  arrow  block  in  Figure  1 )  is  kept  actively 
involved  throughout  the  development  process — providing  key  insights  the  software 
developer  needs  to  successfully  develop  warfighter  capabilities  in  a  sustainable  design  for 
long-term  effectiveness  and  suitability.  The  system  development  activities  are  conducted 
with  superior  understanding  and  clarity,  reducing  scrap  and  rework,  and  saving  cost  and 
schedule.  The  technical  reviews  and  audits  (part  of  the  DoD’s  overarching  SEP)  are 
supported  with  methodologies  that  enhance  both  the  visibility  of  the  necessary  development 
work  as  well  as  the  progress  toward  completing  it. 
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One  of  the  main  goals  in  analyzing  the  scenarios  is  to  discover  key  architectural 
decision  points  that  pose  risks  for  meeting  quality  requirements.  Sensitivity  points  are 
determined,  such  as  real-time  latency  performance  shortfalls  in  target  tracking.  Trade-off 
points  are  also  examined  so  that  TOC  impacts  resulting  from  proposed  trade-offs  can  be 
analyzed.  The  SEI  explained,  “Trade-off  points  are  the  most  critical  decisions  that  one  can 
make  in  an  architecture,  which  is  why  we  focus  on  them  so  carefully”  (Kazman  et  al.,  2000, 
p.  23). 

The  ATAMsm  provides  an  analysis  methodology  that  complements  and  enhances 
many  of  the  key  DoD  acquisition  processes.  It  provides  the  requirements  loop  analysis  in 
the  SEP,  extends  the  user/stakeholder  JCIDS  involvement  through  scenario  development, 
provides  informed  architectural  trade-off  analyses,  and  vastly  improves  the  software 
developer’s  understanding  of  the  quality  requirements  in  context.  Architectural  risk  is 
significantly  reduced,  and  the  software  architecture  presented  at  the  Preliminary  Design 
Review  (PDR)  is  likely  to  have  a  much  higher  probability  of  meeting  the  warfighters’  need  for 
capability,  including  TOC  elements. 

Together,  the  QAW  and  ATAMsm  provide  effective  tools  for  addressing  problem 
areas  common  in  many  DoD  software-intensive  system  developments:  missing  or  vaguely 
articulated  performance  requirements,  significantly  underestimated  software  development 
efforts  (resulting  in  severely  underestimated  schedules  and  budgets),  and  poor 
communication  between  the  software  developer  and  the  government  (both  user  and  PM). 
Both  tools  provide  frameworks  for  more  detailed  requirements  development  and  more 
effective  communication,  but  they  are  just  tools — by  themselves,  they  will  not  replace  the 
need  for  sound  planning,  management  techniques,  and  effort.  Both  the  QAW  and  ATAMsm 
provide  methodologies  for  executing  SEP  requirements  analysis  and  requirements  loop 
functions,  effective  architectural  design  transition  from  user  to  developer,  and  SEP  design 
loop  and  verification  loop  functions  within  the  test-case  development. 

A  significant  product  resulting  from  the  ATAMsm  is  the  development  of  test  cases 
correlating  to  the  use  case,  growth,  and  exploratory  scenarios  developed  and  prioritized. 
Figure  2  depicts  the  progression  from  user-stated  capability  requirements  in  the  JCIDS 
documents  to  the  ATAMsm  scenario  development,  and  finally  to  the  corresponding  test 
cases  developed.  The  linkage  to  the  user  requirements  defined  in  the  JCIDS  documents  is 
very  strong  as  those  documents  drive  the  development  of  the  three  types  of  scenarios,  and, 
in  turn,  the  scenarios  drive  the  development  of  the  use  cases.  The  prioritization  of  the 
scenarios  from  user-stated  KPPs,  Critical  Operational  Issues  (COIs),  and  FMECA  analysis 
flows  to  the  test  cases,  helping  to  create  a  system  test  program  designed  to  focus  on 
effectiveness  and  suitability  tests — culminating  in  the  system  Operational  Test  and 
Evaluation  (OT&E).  FMECA  is  one  of  the  focus  areas  that  will  have  a  dynamic  impact  on 
TOC  analysis  because  it  will  help  identify  software  components  that  need  higher  reliability 
and  back-up  capability.  The  MUIRS  focus  helps  ensure  that  TOC  elements  are  addressed  in 
design  and  test. 
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Figure  2.  Capabilities-Based  ATAMsm  Scenario  Development 

The  traceability  from  user-stated  requirements  through  scenario  development  to  test- 
case  development  provides  a  powerful  communication  and  assessment  methodology.  The 
growth  scenarios  and  resulting  test  cases  are  particularly  suited  for  addressing  and 
evaluating  TOC  design  requirements  because  the  system  evolves  over  its  life  cycle,  which  is 
often  overlooked  in  current  system  development  efforts. 

The  software  developer’s  understanding  of  the  eventual  performance  required  in 
order  to  be  considered  successful  guides  the  design  of  the  architecture  and  every  step  of 
the  software  development,  coding,  and  testing  through  to  the  Full  Operational  Capability 
(FOC)  delivery  and  OT&E.  Coding  and  early  testing  of  software  units  and  configuration 
items  is  much  more  purposeful  due  to  this  level  of  understanding.  The  MUIRS  and  FMECA 
focus  will  help  the  design  process  for  better  TOC  performance. 

The  resulting  test  program  is  very  comprehensive  as  each  prioritized  scenario 
requires  testing  or  other  verification  methodologies  to  demonstrate  how  the  software 
performs  in  each  related  scenario  and  satisfies  the  quality  attributes  borne  of  the  user 
requirements.  The  testing  supports  the  SEP  design  loop  by  verifying  that  the  software 
performs  the  functions  allocated  to  it  and,  in  aggregate,  performs  the  verification  loop 
process  by  demonstrating  that  the  final  product  produces  the  capability  identified  in  the  user 
requirements  through  operational  testing. 

Both  the  QAW  and  ATAMsm  require  the  capturing  of  essential  data  supporting 
decision-making  and  documenting  decisions  made.  These  databases  would  be  best  used  in 
a  collaborative  IT  system,  as  described  in  the  next  section. 

Collaborative  IT  Systems 

Collaborative  IT  tools  are  being  used  today  in  the  private  sector  to  connect  various 
stakeholders — designers,  logisticians,  cost  analysts,  field  service  representatives  (FSRs), 
system  users — who  have  the  need  to  communicate.  Such  tools  could  be  used  to  support 
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current  and  emerging  warfighting  systems.  Collaborative  tools  could  be  adapted  to  address 
reliability  and  ownership  cost  concerns  related  to  warfighting  systems.  Tools  that  facilitate 
improved  communications  would  likely  have  immediate  payoff  in  being  able  to  speed  up 
solutions  to  problems.  For  example,  FSRs  and  users  could  quickly  raise  problems  to 
technical  staff  for  resolution.  Cost  analysts  could  more  quickly  identify  emerging  cost  drivers 
and  initiate  business  case  analyses  (BCAs).  Production  and  quality  technicians  could  rapidly 
learn  of  field  defects  that  are  the  result  of  production  defects.  Other  FSRs  and  users  could 
be  alerted  to  emerging  problems  and  be  armed  with  advance  knowledge  that  might  avert 
impending  failures. 

The  reliability  improvement  process  could  be  enhanced  by  the  use  of  collaborative 
tools,  because  of  the  ease  with  which  LCL  professionals  could  bring  repair  parts  databases 
to  bear  on  design  decisions.  This  would  be  helped  by  Pareto,  that  is,  a  focus  on  the  cost 
drivers  or  reliability  drivers,  especially  the  expensive  items  that  fail  more  often  than 
predicted.  This  approach  could  be  used  up  front  in  pre-acquisition  phases,  too,  by  tying  in 
legacy  databases  that  contain  performance  information  of  similar  or  predecessor  systems. 

Think  of  the  impact  to  BCA.  Cost  estimates  depend  on  solid  cost  databases  that  are 
continually  updated  by  current  systems  in  order  to  identify  major  cost  drivers  that  might  be 
candidates  for  redesign  or  improved  manufacturing  processes  to  achieve  better  reliability 
and  reduced  LCC.  Collaborative  IT  could  contribute  to  the  accuracy  and  completeness  of 
cost  estimates. 

Component  improvements  that  result  from  collaborative  databases  would  pay  off  in 
legacy  systems,  but  might  deliver  a  second  payoff  in  reduced  ownership  cost  of  future 
systems  as  well.  Collaborative  databases  could  be  cross-referenced  in  an  architecture  that 
would  arrange  cost  and  reliability  information  in  system,  subsystem,  or  component 
databases,  enabling  better  cost  estimating  of  emerging  systems.  In  her  2010  article  in  the 
Defense  Acquisition  Review  Journal,  Marti  A.  Roper  discussed  the  need  for  databases  that 
support  acquisition  cost  estimates — down  to  subsystem  or  component  levels,  showing  cost 
ranges.  Such  a  knowledge  base  is  critical  for  the  development  of  follow-on  systems  so  that 
known  cost  drivers  can  be  addressed  for  potentially  significant  LCC  savings  with 
deployment  of  the  replacement  system.  Roper  referred  to  this  as  capabilities-based 
parametric  data  analysis  (2010,  pp.  71-73). 

An  example  of  the  potential  value  of  collaborative  efforts  in  improving  reliability  and 
reducing  TOC  is  the  microwave  tube  on  the  Aegis  program,  developed  in  the  early  1980s. 

The  tubes  were  expensive  to  maintain  (an  estimated  $8.20  per  operating  hour)  and 
ubiquitous  (nearly  30,000  units  in  2010),  and  initial  reliability  numbers  were  lower  than 
expected  (as  low  as  1,300  hours  MTBF).  Through  a  collaborative  effort  between  the  PM, 

NAVSEA,  and  several  commercial  vendors,  design  and  manufacturing  improvements 
increased  the  MTBF  to  40,000-45,000  hours,  drastically  reducing  the  associated  TOC  from 
$8.20  to  $0.45  per  operating  hour  for  all  associated  Naval  combat  systems  (Apte  & 

Dutkowski,  2006,  pp.  3-21). 

Collaborative  IT  tools  could  potentially  be  implemented  through  apps  to  smart 
handheld  devices,  such  as  iPhones,  Androids,  or  Blackberries.  These  devices,  which  are 
ubiquitous  at  systems  commands  and  contractor  design  and  logistics  facilities,  could  be  very 
valuable  and  convenient  for  FSRs,  military  maintenance  personnel,  and  even  users  in  some 
environments. 

Very  possibly,  collaborative  IT  tools  are  in  use,  contributing  to  better  data  and  faster 
solutions  to  service  member  problems  on  legacy  systems.  On  its  face,  the  DoD  needs  to 
embrace  such  tools  to  improve  the  flow  of  technology,  acquisition,  and  logistics  information. 
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Databases 

The  Defense  Acquisition  Management  Information  Retrieval  (DAMIR) — MDAP 
Systems  database  is  a  “virtual”  repository  used  by  the  acquisition  community  and  others  to 
manage  MDAP  and  MAIS  systems  and  to  provide  relevant  information  about  those  systems 
across  the  DoD.  The  database  arrays  Selected  Acquisition  Reports  (SAR),  Defense 
Acquisition  Executive  Summary  (DAES)  reports,  Acquisition  Program  Baselines  (APBs),  and 
SAR  Baselines.  It  contains  other  program  information,  such  as  missions  and  descriptions, 
system  performance,  schedules,  cost  and  funding  (including  operations  and  support  costs), 
Nunn-McCurdy  breaches,  contracts  performance,  and  manufacturing  and  deliveries.  The 
DAMIR  database  contains  some  capability  to  compare  programs  in  terms  of  cost  and 
schedule  performance  and  to  summarize  cost  and  schedule  information  (e.g.,  by  warfighting 
system  or  Service). 

VAMOSC  databases  that  collect  O&S  cost  information  should  be  improved  or 
replaced  for  better  support  of  cost  estimating.  Current  GAO  reports  indicate  that  VAMOSC  is 
inaccurate,  incomplete,  and  internally  inconsistent.  VAMOSC  should  be  able  to  provide  data 
on  similar  or  predecessor  systems,  subsystems,  and  components  in  support  of  programs  in 
development,  in  addition  to  providing  accurate  O&S  cost  performance  for  legacy  systems  in 
their  sustainment  phase. 

Software  component  analysis  and  decision  databases,  like  those  that  would  be 
developed  using  the  QAW  and  ATAMsm  tools,  should  be  required  for  every  software¬ 
intensive  system.  Software  continues  to  be  a  “wildcard”  in  estimating  both  acquisition  costs 
and  O&S  costs,  so  front-end  analyses  must  be  improved,  cataloged,  and  shared  widely 
through  a  collaborative  environment. 

Collaborative  databases  to  gather  enterprise/system/subsystem/component  cost 
information  should  be  established  to  facilitate  collaboration  among  experts  who  are  widely 
dispersed.  One  can  envision  collaborative  IT  systems  being  employed  by  systems 
commands  and  the  DLA.  Such  systems  could  support  national-level  enterprise  requirements 
at  one  end  of  the  spectrum  or  components  at  the  opposite  end.  In  any  case,  collaborative  IT 
systems  could  be  set  up  for  broad  sharing  of  information  that  might  be  useful  to  developers 
of  new  systems,  to  maintainers  of  legacy  systems,  or  to  O&S  cost  analysts  trying  to  improve 
the  performance  of  components  that  are  cost  drivers. 

Conclusions  and  Recommendations:  Major  Thrusts  to  Control  TOC 

Many  of  the  TOC  initiatives  implemented  since  our  TOC  research  report  in  2003  are 
definitely  steps  in  the  right  direction  for  understanding,  assessing,  and,  ultimately,  reducing 
the  TOC  financial  burden.  In  this  research,  we  have  identified  several  areas  that  remain  as 
significant  hindrances  to  effective  TOC  assessment  and  reduction,  including  conflicting 
policy  guidance,  inadequate  or  missing  databases,  and  inadequate  process  controls  for 
software  and  SoS/net-centric  TOC  drivers.  Future  policy  and  guidance  should  address  these 
shortfalls  to  more  fully  address  TOC  issues. 

Controls 

Cost  Estimates 

The  DoD  has  not  yet  demonstrated  its  ability  to  estimate  program  costs  within 
reasonable  confidence  limits.  Estimation  of  developmental  costs  is  challenging  at  best  and 
is  not  yet  well  enough  supported  by  solid  cost  databases.  The  addition  of  O&S  cost 
requirements  makes  sense  from  the  perspective  of  life  cycle  affordability,  but  again,  this 
effort  is  not  supported  by  sufficient  O&S  cost  databases.  The  development  of  SoS  and  net- 
centric  systems  exacerbates  the  cost-estimating  problem  as  system-wide  changes  drive 
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platform  costs,  but  may  not  be  attributable  to  the  platform  absorbing  the  cost.  Platform 
changes  may  also  drive  system-wide  changes,  again  driving  costs  that  are  not  attributable 
to  the  system  level.  While  these  costs  may  not  be  attributable,  we  recognize  that  they  still 
need  to  be  tracked  so  that  they  can  be  estimated  in  future  developments  and  so  that  root- 
cause  analyses  can  be  applied  to  help  eliminate  the  sources  in  the  future. 

Certifications  at  MS  A  and  MS  B 

The  certifications  at  Milestones  A  and  B,  along  with  the  attention  of  the  Director 
CAPE,  undoubtedly  bring  attention  and  scrutiny  to  program  cost  estimates  and  concerns 
regarding  program  affordability  in  the  context  of  the  larger  warfighting  portfolio.  The  mandate 
for  cost  certificates  is  a  major  improvement,  as  compared  to  our  2003  research.  Cost 
certificates  are  a  necessary  forcing  function  to  push  the  DoD  toward  more  reliable  cost 
estimating.  Again,  SoS  and  net-centric  system  development  may  add  certification 
challenges  as  the  associated  costs  are  typically  not  foreseeable,  and  attributing  the  costs  to 
a  specific  PM  may  be  difficult. 

Changes  to  Nunn-McCurdy  to  Include  an  O&S  Cost  Metric 

Unquestionably,  Nunn-McCurdy  requirements  have  become  more  demanding  and 
onerous.  As  challenging  as  acquisition  costs  (APUC  and  PAUC)  are,  they  are  not  the 
correct  metrics  when  viewed  from  a  life  cycle  cost  perspective.  Nunn-McCurdy  metrics  need 
to  evolve  into  measures  of  life  cycle  cost,  including  O&S  cost  portion  (e.g.,  average  O&S 
cost  per  system  per  hour  or  average  O&S  cost  per  system  per  mile).  To  do  otherwise  is  to 
encourage  poor  system  development  choices  that  may  add  to  life  cycle  cost  rather  than 
constrain  it. 

Mandated  Reviews 

Moving  the  PDR  Assessment  to  precede  or  coincide  with  Milestone  B,  as  mandated 
in  WSARA,  should  improve  decision-making.  That  is,  required  warfighter  capabilities, 
technological  maturity,  affordable  resources,  and  available  schedule  must  be  compatible 
with  the  system  specification  at  Milestone  B.  This  cannot  be  properly  assured  without 
completion  of  the  preliminary  design  because  PDR  supports  preparation  of  resource  and 
schedule  estimates.  To  that  end,  we  recommend  that  software-intensive  systems  employ 
the  SEI’s  QAW  and  ATAMsm  process  tools  (or  similar-type  processes)  to  accomplish  the 
following:  more  fully  define  derived  and  implied  software-related  requirements;  improve  the 
software  developer’s  understanding  of  how  the  warfighters  use  and  maintain  the  system; 
understand  how  the  system  is  likely  to  be  changed,  modified,  or  made  interoperable  over  its 
life  cycle;  and  improve  the  developer’s  understanding  of  the  performance  the  warfighter 
expects  under  stressful  or  unusual  operating  scenarios.  These  process  tools  should  vastly 
improve  the  reliability  of  information  resulting  from  the  PDR  with  regard  to  the  software 
components. 

Technological  Maturity 

The  Technology  Readiness  Assessment  Deskbook  was  published  in  2003  and  has 
greatly  clarified  understanding  of  technological  maturity,  yet  it  is  difficult  to  apply  to  software 
development.  The  DoD  has  a  long  track  record  of  moving  into  detailed  design  after 
Milestone  B  without  the  necessary  maturity  of  technology  to  complete  the  system  design. 

The  result  is  almost  always  program  delays  and  substantial  cost  growth.  Lack  of 
technological  maturity  is  one  of  the  major  causes  of  cost  growth  and  reflects  the  importance 
of  Knowledge  Point  1 ,  as  described  by  the  GAO.  Because  software  development  defies 
early  maturity  estimation,  it  must  be  considered  separately  and  include  the  maturity 
evaluations  of  the  software  developer  (CMMI  or  equivalent),  as  well  as  the  maturity 
evaluations  of  the  materiel  developer/PM  (SA-CMM  or  equivalent). 
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Today,  we  have  a  useable  template  to  discuss  and  reach  a  common  understanding 
of  technological  maturity;  we  know  the  importance  of  technological  maturity;  we  have  a 
mandated  certification — in  law  and  regulation — to  assure  the  intersection  of  technological 
maturity,  affordability,  available  budget,  and  schedule.  The  DoD  knows  the  elements  of 
knowledge  that  are  necessary  for  sound  decision-making  to  launch  development  of  a  new 
warfighting  system.  This  also  applies  to  COTS  or  GOTS  software,  but  software  development 
depends  on  assessing  the  maturity  of  the  developer  and  the  PM  office,  as  stated  previously. 

Navy  Gate  Reviews 

The  DoD  should  require  gate  reviews  for  use  by  all  the  Services.  Gate  reviews 
provide  for  oversight  and  governance  of  MDAP  life  cycle  cost.  These  reviews  establish  a 
process  to  bring  attention  to  ownership  cost  throughout  the  developmental  cycle  of 
warfighting  systems.  In  a  wider  sense,  gate  reviews  provide  a  forum  for  lessons  learned 
regarding  TOC.  While  emphasizing  affordability  through  the  developmental  and  production 
phases  of  individual  warfighting  systems,  gate  reviews  provide  the  opportunity  to  balance 
the  resources  provided  among  capability  portfolios,  and  potentially  to  assist  in  balancing 
resources  across  all  of  the  department’s  family  of  capability  portfolios. 

Configuration  Steering  Boards 

The  opportunity  to  grow  requirements  for  ongoing  programs  that  are  beyond 
Milestone  B  has  been  largely  taken  away  from  the  user  community  and  placed  into  the 
hands  of  each  Service’s  Configuration  Steering  Board.  This  is  likely  to  curtail  major  cost 
increases  in  programs  and  encourages  cost  reductions  based  on  PM  recommendations  in 
program  requirements  and  within  program  objectives.  Congressional  language  on  changes 
to  user  requirements  has  been  accommodated  in  the  most  recent  version  of  DoDI  5000.02, 
dated  December  2,  2008.  Implementation  of  this  guidance  entails  a  major  change  in  culture; 
whether  it  is  successful  in  reducing  ownership  cost  will  be  shown  over  time. 

Performance-Based  Logistics 

The  DoD  is  very  familiar  with  the  demands  of  sustainment — but  the  OSD  has  not 
insisted  on  proper  planning  and  implementation  of  affordable  sustainment.  The  OSD  has  not 
focused  enough  on  the  metrics  that  indicate  success  of  warfighting  systems  or  on  the  cost  to 
achieve  required  metrics.  Instead,  focus  has  been  on  commodity  management,  with  the 
DLA  being  a  prime  example,  where  metrics  have  reflected  performance  of  the  support 
organization,  but  not  weapon  system  readiness. 

PBL  must  be  applied  more  widely,  such  that  non-PBL  systems  should  be  an  unusual 
occurrence.  PBL  requirements  initially  should  be  analyzed  vertically  by  an  individual  system 
such  that  the  warfighting  system  is  able  to  achieve  its  mission  and  is  affordable.  However, 
PBL  arrangements  also  should  be  analyzed  horizontally  to  take  advantage  of  economic 
quantities  and  other  efficiencies  that  might  be  provided  by  using  common  support  systems. 
PBL  metrics  also  should  be  devised  to  reflect  the  individual  warfighting  system  (i.e.,  vertical) 
and  the  broader  support  system  or  enterprise  (i.e.,  horizontal). 
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Total  Ownership  Cost — a  Decade  into  the  21st  Century 

2003  -  Where  We  Were 


•  Positive 

-  Encouraged  experimentation 

-  Come  up  with  new  approaches 

•  Negative 

-  Leadership  wasn’t  heavily  involved  in  affordability  - 
not  speaking  with  one  voice 

-  Lack  of  Discipline  (e.g.,  technological  maturity) 

-  Lacked  risk  assessment  tools  (TRLs,  MRLs,  SMLs) 
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Total  Ownership  Cost — a  Decade  into  the  21st  Century 

Today  -  Where  We  Are 


•  Cost  estimating 

-  Certifications 

-  Nunn-McCurdy 

-  Cost  Databases 

-  Affordability  Slices  of  Mission  Areas 

•  Collaborative  IT 

•  Mandated  reviews  -  MS  B  (KP-1 ),  CDR-A  (KP-2),  MS  C  (KP-3) 

•  Navy  Gate  Reviews  (affordability) 

•  Configuration  Steering  Boards  (counter  to  requirements  creep) 

•  Product  Support  Manager  -  Performance  Based  Logistics 
(affordable  logistics) 
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Total  Ownership  Cost — a  Decade  into  the  21st  Century 

Opportunities 

•  Cost  estimating  impediments 

-  Statistical  Confidence  Levels 

-  Useful  Cost  Databases  -  support  early  cost 
estimates? 

-  Nunn-McCurdy  Breaches  using  the  wrong  metrics 

-  Cost  vs.  Affordability 


•  Collaborative  IT 

-  Are  the  right  stakeholders  involved  in  the 
conversation? 
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Total  Ownership  Cost — a  Decade  into  the  21st  Century 

Summary 


•  Mandated  discipline 

•  Bureaucracy 

•  Selective  lack  of  tools 

•  Need  to  move  to  self-motivated  discipline 
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Software  Intensive  Systems  &  TOC 


•  Poor  SW  size  &  complexity  estimates  lead  to  understated  SW 
f  O&S  cost  estimates 

•  Requirements  progression  from  user  ‘Capability  Need’  through 
PM  ‘Performance  Spec’  to  contractor  ‘System  Design’  invites 
requirements  interpretation 

•  Interpretation  leads  to  vague  or  missed  requirements 

[•  Vague/missing  requirements  lead  to  poor  SW  size  & 
complexity  estimates 


•  Repeat  as  necessary! 
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Bridging  the  SW  Requirements  Gap 


•  The  immature  SW  engineering  environment  is  incapable  of 
satisfying  unstated  requirements  -  especially  supportability 
performance  gaps 

•  Requirements  gap  analysis  essential  for  attaining  SW 
supportability  performance  -  MUIRS  Analysis: 
Maintainability,  Upgradability,  Interoperability,  Reliability, 
Safety  &  Security 

•  Goal:  Develop  complete,  well  defined  inventory  of 
requirements,  including  stated,  derived,  and  implied 


•  Tools:  MUIRS  Analysis  &  SEI’s  Quality  Attribute  Workshop 
(QAW) 
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SW  Design  -  The  Key  to  O&S  Performance 


•  Must  drive  the  design  for  supportability  performance 

•  Starts  with  a  complete  inventory  of  requirements,  including 
supportability  requirements  resulting  from  a  Logistics 
Supportability  Analysis  -  MUIRS 


•  SW  developer  needs  to  know  requirements  in  context  -  How 
will  system  be  used  &  maintained?  In  what  environments? 
What  is  the  priority  of  essential  functions  &  enhancing 
functions?  How  should  it  operate  when  stressed?  What  is 
the  expected  exception  handling,  fault  tolerance,  and 
recovery  techniques?  How  will  performance  be  verified? 
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SW  Design  continued 


•  User  involvement  in  the  SW  design  process  is  critical  -  they 
must  develop  scenarios  for: 

-  Use  Cases:  Including  MUIRS  focus  for  supportability 

-  Growth:  Anticipated  changes  over  the  life  cycle 

-  Exploratory:  Expected  performance  when  stressed,  including  FMECA 
prioritization  of  functionality/recovery 

•  Goal:  Ensure  SW  developer  understands  warfighter 
expectations  before  system  is  designed 


•  Tools:  SEI’s  Architectural  Tradeoff  Analysis  Methodology 
(ATAM  «") 
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QAW  &  ATAM  Integration  into  SW  Lifecycle  Management 
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Summary:  Improving  SW  TOC  Performance 


•  Break  the  cycle:  Poor  requirements/designs  =  difficult  and 
costly  SW  sustainment 

•  Complete  the  inventory  of  derived  and  implied  SW 
supportability  requirements  with  MUIRS  analysis 

-  Tools:  MUIRS  Analysis  technique  and  SEI’s  QAW 

•  Drive  the  system  design  for  improved  supportability 
performance,  critical  for  Software 

-  Tool:  SEI’s  ATAM  sm 


•  Ensure  test  program  includes  supportability  performance 
testing,  stress  testing,  fault  handling,  and  recovery 
techniques 

-  Tool:  SEI’s  ATAM  sm  -  T  ' 
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